Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Airflow 2 support #643

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open

Airflow 2 support #643

wants to merge 26 commits into from

Conversation

dataops-sre
Copy link

TL;TR

This repo seems no more maintained, I forked the original repo for airflow 2 support, check here. do:
docker pull dataopssre/docker-airflow2:2.1.2

What does this PR do?

Add airflow2 support

Motivation

Airflow is been updated to version 2 and release its official docker image, you can also find bitnami airflow image. Nevertheless, puckel's image is still interesting, in the market none of providers offer an Airflow run with LocalExecutor with scheduler in one container, it is extremely usefull when to deploy a simple Airflow to an AWS EKS cluster. With Kubernetes you can resolve Airflow scablity issue by using uniquely KubernetesPodOpetertor in your dags, then we need zero computational power for airflow, it serves pure purpose of scheduler, seperate scheduler and webserver into two different pods is a bit problematic on AWS EKS cluster, we want to keep dags and logs into a Persistant volume, but AWS has some limitation for EBS volume multi attach, which means webserver and scheduler pod has to be scheduled on the same EKS node, it is a bit annoying. Thus puckel's airflow startup script is usefull.

Additional Notes

what this fork do :

  • Disactive by default the login screen in Airflow 2
  • Improve current script to only take into account Airflow environment variables
  • Make sure docker compose files works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants